Image Acquisition Method and Apparatus

ABSTRACT

An image acquisition sensor of a digital image acquisition apparatus is coupled to imaging optics for acquiring a sequence of images. Images acquired by the sensor are stored. A motion detector causes the sensor to cease capture of an image when the degree of movement in acquiring the image exceeds a threshold. A controller selectively transfers acquired images for storage. A motion extractor determines motion parameters of a selected, stored image. An image re-constructor corrects the selected image with associated motion parameters. A selected plurality of images nominally of the same scene are merged and corrected by the image re-constructor to produce a high quality image of the scene.

PRIORITY

This application is a Continuation of U.S. patent application Ser. No.11/753,098, filed May 24, 2007, which claims the benefit of priorityunder 35 USC §119 to U.S. provisional patent application No. 60/803,980,filed Jun. 5, 2006, and to U.S. provisional patent application No.60/892,880, filed Mar. 5, 2007, which are incorporated by reference.

The present invention relates to an image acquisition method andapparatus, in particular, the invention addresses the problems ofacquisition device or subject movement during image acquisition.

BACKGROUND OF THE INVENTION

The approach to restoring an acquired image which is degraded or uncleareither due to acquisition device or subject movement during imageacquisition, divides in two categories:

-   -   Deconvolution where an image degradation kernel, for example, a        point spread function (PSF) is known; and    -   Blind deconvolution where motion parameters are unknown.

Considering blind deconvolution (which is the most often case in realsituations), there are two main approaches:

-   -   identifying motion parameters, such as PSF separately from the        degraded image and using the motion parameters later with any        one of a number of image restoration processes; and    -   incorporating the identification procedure within the        restoration process. This involves simultaneously estimating the        motion parameters and the true image and it is usually done        iteratively.

The first blind deconvolution approach is usually based on spectralanalysis. Typically, this involves estimating the PSF directly from thespectrum or Cepstrum of the degraded image. The Cepstrum of an image isdefined as the inverse Fourier transform of the logarithm of thespectral power of the image. The PSF (point spread function) of an imagemay be determined from the cepstrum, where the PSF is approximatelylinear. It is also possible to determine, with reasonable accuracy, thePSF of an image where the PSF is moderately curvilinear. Thiscorresponds to even motion of a camera during exposure. It is known thata motion blur produces spikes in the Cepstrum of the degraded image.

So, for example, FIG. 5 a shows an image of a scene comprising a whitepoint on a black background which has been blurred to produce the PSFshown. (In this case, the image and the PSF are the same, however, itwill be appreciated that for normal images this is not the case.) FIG. 5b shows the log of the spectrum of the image of FIG. 5 a, and thisincludes periodic spikes in values in the direction 44 of the PSF. Thedistance from the center of spectrum to the nearest large spike value isequal to the PSF size. FIG. 5 c shows the Cepstrum of the image, wherethere is a spike 40 at the centre and a sequence of spikes 42. Thedistance between the center 40 and the first spike 42 is equal to thePSF length.

Techniques, for example, as described at M. Cannon “Blind Deconvolutionof Spatially Invariant Image Blurs with Phase” published in IEEETransactions on Acoustics, Speech, and Signal Processing, Vol. ASSP-24,NO. 1, February 1976 and refined by R. L. Lagendijk, J. Biemond in“Iterative Identification and Restoration of Images”, Kluwer AcademicPublishers, 1991 involve searching for those spikes in a Cepstrum,estimating the orientation and dimension of the PSF and, then,reconstructing the PSF from these parameters. This approach is fast andstraight-forward, however, good results are usually generally achievedonly for uniform and linear motion or for out of focus images. This isbecause for images subject to non-uniform or non-linear motion, thelargest spikes are not always most relevant for determining motionparameters.

A second blind deconvolution approach involves iterative methods,convergence algorithms, and error minimization techniques. Usually,acceptable results are only obtained either by restricting the image toa known, parametric form (an object of known shape on a dark backgroundas in the case of astronomy images) or by providing information aboutthe degradation model. These methods usually suffer from convergenceproblems, numerical instability, and extremely high computation time andstrong artifacts.

A CMOS image sensor may be built which can capture multiple images withshort exposure times (SET images) as described in “A Wide Dynamic RangeCMOS Image Sensor with Multiple Short-Time Exposures”, Sasaki et al,IEEE Proceedings on Sensors, 2004, 24-27 Oct. 2004 Page(s):967-972 vol.2.

Multiple blurred and/or undersampled images may be combined to yield asingle higher quality image of larger resolution as described in“Restoration of a Single Superresolution Image from Several Blurred,Noisy and Undersampled Measured Images”, Elad et al, IEEE Transactionson Image Processing, Vol. 6, No. 12, December 1997.

SUMMARY OF THE INVENTION

A digital image acquisition apparatus is provided. An image acquisitionsensor is coupled to imaging optics for acquiring a sequence of images.An image store is for storing images acquired by the sensor. A motiondetector is for causing the sensor to cease capture of an image when adegree of movement in acquiring the image exceeds a threshold. Acontroller selectively transfers the image acquired by the sensor to theimage store. A motion extractor determines motion parameters of aselected image stored in the image store. An image re-constructorcorrects a selected image with associated motion parameters. An imagemerger is for merging selected images nominally of the same scene andcorrected by the image re-constructor to produce a high quality image ofthe scene.

The motion extractor may be configured to estimate a point spreadfunction (PSF) for the selected image. The motion extractor may beconfigured to calculate a Cepstrum for the selected image, identify oneor more spikes in the Cepstrum, and select one of the spikes in theCepstrum as an end point for the PSF. The extractor may be configured tocalculate a negative Cepstrum, and to set points in the negativeCepstrum having a value less than a threshold to zero.

The image store may include a temporary image store, and the apparatusmay also include a non-volatile memory. The image merger may beconfigured to store the high quality image in the non-volatile memory.

The motion detector may include a gyro-sensor or an accelerometer, orboth.

A further digital image acquisition apparatus is provided. An imageacquisition sensor is coupled to imaging optics for acquiring a sequenceof images. An image store is for storing images acquired by said sensor.A motion detector causes the sensor to cease capture of an image whenthe degree of movement in acquiring the image exceeds a first threshold.One or more controllers cause the sensor to restart capture when adegree of movement is less than a given second threshold, andselectively transfer images acquired by the sensor to the image store. Amotion extractor determines motion parameters of a selected image storedin the image store. An image re-constructor corrects a selected imagewith associated motion parameters. An image merger merges selectedimages nominally of the same scene and corrected by the imagere-constructor to produce a high quality image of the scene.

A first exposure timer may store an aggregate exposure time of thesequence of images. The apparatus may be configured to acquire thesequence of images until the aggregate exposure time of at least astored number of the sequence of images exceeds a predetermined exposuretime for the high quality image. A second timer may store an exposuretime for a single image. An image quality analyzer may analyze a singleimage. The apparatus may be configured to dispose of an image having aquality less than a given threshold quality and/or having an exposuretime less than a threshold time.

The image merger may be configured to align the images prior to mergingthem. The first and second thresholds may include threshold amounts ofmotion energy.

An image capture method with motion elimination is also provided. Anoptimal exposure time is determined for the image. A sequence ofconsecutive exposures is performed, including:

(i) exposing intermediate images until either the optimal exposure timeis reached or motion is detected beyond an excessive movement threshold;and

(ii) discarding images that have insufficient exposure times or thatexhibit excessive movement;

(iii) storing non-discarded intermediate images for further imagerestoration, including:

(iv) performing motion de-blurring on non-discarded intermediate images;

(v) calculating a signal to noise ratio and, based on the calculating,performing exposure enhancement on the non-discarded images;

(vi) performing registration between restored intermediate images;

(vii) assigning a factor to each of the restored images based on qualityof restoration, signal to noise ratio or overall exposure time, orcombinations thereof; and

(viii) merging the restored images based on a weighted contribution asdefined by said factor.

An aggregate exposure time of a sequence of images may be stored. Thesequence of images may be acquired until the aggregate exposure time ofat least a stored number of images exceeds a predetermined exposure timefor a high quality image. An exposure time may be stored for a singleimage, and/or an image quality may be analyzed for a single image. Animage may be disposed of that has an exposure time less than a thresholdtime and/or a quality less than a given threshold quality.

The merging may include aligning each restored image. A threshold mayinclude a threshold amount of motion energy.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will now be described by way of example, with reference tothe accompanying drawings, in which:

FIG. 1 illustrates schematically a digital image acquisition apparatusaccording to an embodiment.

FIGS. 2 a-2 b illustrate (a) a PSF for a single image and (b) the PSFsfor three corresponding SET images acquired according to the anembodiment.

FIGS. 3 a-3 c illustrate how blurring of partially exposed images canreduce the amount of motion blur in the image.

FIG. 4 illustrates the generation of a PSF for an image acquired inaccordance with an embodiment;

FIGS. 5 a-5 e illustrate sample images/PSFs and their correspondingCepstrums.

FIG. 6 illustrates an estimate of a PSF constructed according to anembodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

An image acquisition system is provided in accordance with an embodimentwhich incorporates a motion sensor and utilizes techniques to compensatefor motion blur in an image.

One embodiment is a system that includes the following:

(1) the image acquisition apparatus comprises a imaging sensor, whichcould be CCD, CMOS, etc., hereinafter referred to as CMOS;

(2) a motion sensor (Gyroscopic, Accelerometer or a combinationthereof);

(3) a fast memory cache (to store intermediate images); and

(4) a real-time subsystem for determining the motion (PSF) of an image.Such determination may be done in various ways. One preferred method isdetermining the PSF based on the image cepstrum.

In addition, the system can include a correction component, which mayinclude:

-   -   (a) a subsystem for performing image restoration based on the        motion PSF,    -   (b) an image merging subsystem to perform registration of        multi-images and merging of images or part of images    -   (c) a CPU for directing the operations of these subsystems.

In certain embodiments some of these subsystems may be implemented infirmware and executed by the CPU. In alternative embodiments it may beadvantageous to implement some, or indeed all of these subsystems asdedicated hardware units. Alternatively, the correction stage may bedone in an external system to the acquisition system, such as a personalcomputer that the images are downloaded to.

In one embodiment, the ceptrum may include the Fourier transform of thelog-magnitude spectrum: fFt(ln(|fFt(window.signal)|)).

In a preferred embodiment the disclosed system is implemented on adual-CPU image acquisition system where one of the CPUs is an ARM andthe second is a dedicated DSP unit. The DSP unit has hardware subsystemsto execute complex arithmetical and Fourier transform operations whichprovides computational advantages for the PSF extraction.

Image Restoration and Image Merging Subsystems

In a preferred embodiment, when the acquisition subsystem is activatedto capture an image it executes the following initialization steps: (i)the motion sensor and an associated rate detector are activated; (ii)the cache memory is set to point to the first image storage block; (iii)the other image processing subsystems are reset and (iv) the imagesensor is signaled to begin an image acquisition cycle and (v) acount-down timer is initialized with the desired exposure time, acount-up timer is set to zero, and both are started.

In a given scene an exposure time is determined for optimal exposure.This will be the time provided to the main exposure timer. Another timeperiod is the minimal-accepted-partially exposed image. When an image isunderexposed (the integration of photons on the sensor is not complete)the signal to noise ratio is reduced. Depending on the specific device,the minimal accepted time is determined where sufficient data isavailable in the image without the introduction of too much noise. Thisvalue is empirical and relies on the specific configuration of thesensor acquisition system.

The CMOS sensor proceeds to acquire an image by integrating the lightenergy falling on each sensor pixel. If no motion is detected, thiscontinues until either the main exposure timer counts down to zero, atwhich time a fully exposed image has been acquired. However, in thisaforementioned embodiment, the rate detector can be triggered by themotion sensor. The rate detector is set to a predetermined threshold.One example of such threshold is one which indicates that the motion ofthe image acquisition subsystem is about to exceed the threshold of evencurvilinear motion which will allow the PSF extractor to determine thePSF of an acquired image. The motion sensor and rate detector can bereplaced by an accelerometer and detecting a +/− threshold level. Thedecision of what triggers the cease of exposure can be made on inputform multiple sensor and or a forumale trading of non-linear motion andexposure time.

When the rate detector is triggered then image acquisition by the sensoris halted. At the same time the count-down timer is halted and the valuefrom the count-up timer is compared with a minimum threshold value. Ifthis value is above the minimum threshold then a useful SET image wasacquired and sensor read-out to memory cache is initiated. The currentSET image data may be loaded into the first image storage location inthe memory cache, and the value of the count-up timer (exposure time) isstored in association with the image. The sensor is then re-initializedfor another short-time image acquisition cycle, the count-up timer iszeroed, both timers are restarted and a new image acquisition isinitiated.

If the count-up timer value is below the minimum threshold then therewas not sufficient time to acquire a valid short-time exposure and dataread-out form the sensor is not initiated. The sensor is re-initializedfor another short-time exposure, the value in the count-up timer isadded to the count-down timer (thus restoring the time counted downduring the acquisition cycle), the count-up timer is re-initialized,then both timers are restarted and a new image acquisition is initiated.

This process repeats itself until in total the exposure exceeds theneeded optimal integration time. If for example in the second SET imagereaches full term of exposure, it will then become the final candidate,with no need to perform post processing integration. If however, nosingle image exceeds the optimal exposure time, an integration isperformed.

This cycle of acquiring another short-time image continues until thecount-down timer reaches zero—in a practical embodiment the timer willactually go below zero because the last short-time image which isacquired must also have an exposure time greater than the minimumthreshold for the count-up timer. At this point there should be Nshort-time images captured and stored in the memory cache. Each of theseshort-time images will have been captured with an curvilinearmotion-PSF. The total sum of N may exceeds the optimal exposure time,which in this case the “merging system will have more images or moredata to choose from overall.

After a sufficient exposure is acquired it is now possible in apreferred embodiment to recombine the separate short-term exposureimages as follows:

(i) each image is processed by a PSF extractor which can determine thelinear or curvilinear form of the PSF which blurred the image;

(ii) the image is next passed onto an image re-constructor which alsotakes the extracted PSF as an input; this reconstructs each short-timeimage in turn. Depending on the total exposure time, this image may alsogo through exposure enhancement which will increase its overallcontribution to the final image. Of course, the decision whether toboost up the exposure is a tradeoff between the added exposure and thepotential introduction of more noise into the system. The decision isperformed based on the nature of the image data (highlight, shadows,original exposure time) as well as the available SET of imagesaltogether. In a pathological example if only a single image isavailable that only had 50% exposure time, it will need to be enhancedto 2× exposure even at the risk of having some noise. If however, twoimages exist each with 50% exposure time, and the restoration isconsidered well, no exposure will be needed. Finally, themotion-corrected and exposure corrected images are passed it onto; and

(iii) the image merger; the image merger performs local and globalalignment of each short-term image using techniques which are well-knownto those skilled in the arts of super-resolution; these techniques alloweach deblurred short-time image to contribute to the construction of ahigher resolution main image.

This approach has several advantages including:

(1) the number of SET is kept to a minimum; if the motion throughout anexposure is constant linear or curvilinear motion then only a singleimage need be captured;

(2) the decision of who at images are used to create the Final image aredetermined post processing thus enabling more flexibility in determiningthe best combination, where the motion throughout an exposure is mostlyregular, but some rapid deviations appear in the middle the inventionwill effectively “skip over” these rapid deviations and a useful imagecan still be obtained; this would not be possible with a conventionalimage acquisition system which employed super-resolution techniquesbecause the SET images are captured for a fixed time interval;

(3) where the image captured is of a time frame that is too small, thisportion can be discarded;

FIG. 1 a shows an Original PSF and FIG. 1 b shows partial PSFs.Referring now to FIG. 1, which illustrates a digital image acquisitionapparatus 100 according to a preferred embodiment of the presentinvention, the apparatus 100 comprises a CMOS imaging sensor 105 coupledto camera optics 103 for acquiring an image.

The apparatus includes a CPU 115 for controlling the sensor 105 and theoperations of sub-systems within the apparatus. Connected to the CPU 115are a motion sensor 109 and an image cache 130. Suitable motion sensorsinclude a gyroscopic sensor (or a pair of gyro sensors) that measuresthe angular velocity of the camera around a given axis, for example, asproduced by Analog Devices' under the part number ADXRS401.

In FIG. 1, a subsystem 131 for estimating the motion parameters of anacquired image and a subsystem 133 for performing image restorationbased on the motion parameters for the image are shown coupled to theimage cache 130. In the embodiment, the motion parameters provided bythe extractor sub-system 131 comprise an estimated PSF calculated by theextractor 131 from the image Cepstrum.

An image merging subsystem 135 connects to the output of the imagerestoration sub-system 133 to produce a single image from a sequence ofone or more de-blurred images.

In certain embodiments some of these subsystems of the apparatus 100 maybe implemented in firmware and executed by the CPU; whereas inalternative embodiments it may be advantageous to implement some, orindeed all of these subsystems as dedicated hardware units.

So for example, in a preferred embodiment, the apparatus 100 isimplemented on a dual-CPU system where one of the CPUs is an ARM Coreand the second is a dedicated DSP unit. The DSP unit has hardwaresubsystems to execute complex arithmetical and Fourier transformoperations, which provides computational advantages for the PSFextraction 131, image restoration 133 and image merging 135 subsystems.

When the apparatus 100 is activated to capture an image, it firstlyexecutes the following initialization steps:

-   -   (i) the motion sensor 109 and an associated rate detector 108        are activated;    -   (ii) the cache memory 130 is set to point to a first image        storage block 130-1;    -   (iii) the other image processing subsystems are reset;    -   (iv) the image sensor 105 is signaled to begin an image        acquisition cycle; and    -   (v) a count-down timer 111 is initialized with the desired        exposure time, a count-up timer 112 is set to zero, and both are        started.

The CMOS sensor 105 proceeds to acquire an image by integrating thelight energy falling on each sensor pixel; this continues until eitherthe main exposure timer counts 111 down to zero, at which time a fullyexposed image has been acquired, or until the rate detector 108 istriggered by the motion sensor 109. The rate detector is set to apredetermined threshold which indicates that the motion of the imageacquisition subsystem is about to exceed the threshold of evencurvilinear motion which would prevent the PSF extractor 131 accuratelyestimating the PSF of an acquired image.

In alternative implementations, the motion sensor 109 and rate detector108 can be replaced by an accelerometer (not shown) and detecting a +/−threshold level. Indeed any suitable subsystem for determining a degreeof motion energy and comparing this with a threshold of motion energycould be used.

When the rate detector 108 is triggered, then image acquisition by thesensor 105 is halted; at the same time the count-down timer 111 ishalted and the value from the count-up timer 112 is compared with aminimum threshold value. If this value is above the minimum thresholdthen a useful short exposure time (SET) image was acquired and sensor105 read-out to memory cache 130 is initiated; the current SET imagedata is loaded into the first image storage location in the memorycache, and the value of the count-up timer (exposure time) is stored inassociation with the SET image.

The sensor 105 is then re-initialized for another SET image acquisitioncycle, the count-up timer is zeroed, both timers are restarted and a newimage acquisition is initiated.

If the count-up timer 112 value is below the minimum threshold, thenthere was not sufficient time to acquire a valid SET image and dataread-out from the sensor is not initiated. The sensor is re-initializedfor another short exposure time, the value in the count-up timer 112 isadded to the count-down timer 111 (thus restoring the time counted downduring the acquisition cycle), the count-up timer is re-initialized,then both timers are restarted and a new image acquisition is initiated.

This cycle of acquiring another SET image 130-n continues until thecount-down timer 111 reaches zero. Practically, the timer will actuallygo below zero because the last SET image which is acquired must alsohave an exposure time greater than the minimum threshold for thecount-up timer 112. At this point, there should be N short-time imagescaptured and stored in the memory cache 130. Each of these SET imageswill have been captured with a linear or curvilinear motion-PSF.

FIGS. 2 a-2 b illustrates Point Spread Functions (PSF). FIG. 2( a) showsthe PSF of a full image exposure interval; and FIG. 2( b) shows how thisis split into five SET-exposures by the motion sensor. In FIG. 2, boxes(1) through (5) are shown, and:

(1) will be used and with the nature of the PSF it has high probabilityof good restoration and also potential enhancement using gain;(2) can be well restored;(3) will be discarded as too short of an integration period;(4) will be discarded having a non-curvliniar motion; and(5) can be used for the final image.

So for example, while a single image captured with a full-exposureinterval might have a PSF as shown in FIG. 2( a), a sequence of 3 imagescaptured according to the above embodiment, might have respective PSFsas shown in FIG. 2( b). It will be seen that the motion for each ofthese SET image PSFs more readily lends the associated images tode-blurring than the more complete motion of FIG. 2( a).

After a sufficient exposure is acquired, it is now possible to recombinethe separate SET images 130-1 . . . 130-N as follows:

-   (i) each image is processed by the PSF extractor 131 which estimates    the PSF which blurred the SET image;-   (ii) the image is next passed onto the image re-constructor 133    which as well as each SET image takes the corresponding estimated    PSF as an input; this reconstructs each SET image in turn and passes    it onto the image merger 135;-   (iii) the image merger 135 performs local and global alignment of    each SET image using techniques which are well-known to those    skilled in the art of super-resolution. These techniques allow each    de-blurred SET image to contribute to the construction of a higher    resolution main image which is then stored in image store 140. The    image merger may during merging decide to discard an image where it    is decided it is detrimental to the final quality of the merged    image; or alternatively various images involved in the merging    process can be weighted according to their respective clarity.    This approach has several benefits over the prior art:-   (i) the number of SET images is kept to a minimum; if the motion    throughout an exposure is constant linear or curvilinear motion then    only a single image needs to be captured;-   (ii) where the motion throughout an exposure is mostly regular, but    some rapid deviations appear in the middle, the embodiment will    effectively “skip over” these rapid deviations and a useful image    can still be obtained. This would not be possible with a    conventional image acquisition system which employed    super-resolution techniques, because the SET images are captured for    a fixed time interval.

Although the embodiment above could be implemented with a PSF extractor131 based on conventional techniques mentioned in the introduction,where a PSF involves slightly curved or non-uniform motion, the largestspikes may not always be most relevant for determining motionparameters, and so conventional approaches for deriving the PSF even ofSET images such as shown in FIG. 2( b) may not provide satisfactoryresults.

Thus, in a particular implementation of the present invention, the PSFextractor 131 rather than seeking spikes in a Cepstrum, seeks largeregions around spikes in the Cepstrum of an image using a region-growingalgorithm. This is performed by inspecting candidate spikes in theCepstrum, using region growing around these candidates and thendiscriminating between them. Preferably, the candidate spike of thelargest region surrounding a candidate spike will be the point chosen asthe last point of the PSF.

It can be seen from FIGS. 3 a-3 c that blurring the partially exposedimage reduces the amount of motion blur in the image. FIG. 3( a) showsan original image. FIG. 3( b) illustrates blurring with full PSF. FIG.3( c) illustrates reconstructed image from 3 SET images using individualPSFs.

Referring to FIG. 3, an SET image 130-1 . . . 130-N is represented inthe RGB space (multi-channel) or as a gray-scale (“one-channel”). TheCepstrum may be computed on each color channel (in the case ofmulti-channel image) or only on one of them and so, by default, theCepstrum would have the size of the degraded image. In the preferredembodiment, the Fourier transform is performed, step 32 only on thegreen channel. It will also be seen that, for processing simplicity, theresults are negated to provide a negative Cepstrum for later processing.

In variations of the embodiment, the Cepstrum may be computed:

-   -   on each channel and, afterwards, averaged; or    -   on the equivalent gray image.

After computing the negative Cepstrum, the blurred image 130 is notnecessary for the extractor 131 and can be released from memory or forother processes. It should also be seen that as the Cepstrum issymmetrical towards its center (the continuous component), only one halfis required for further processing.

As discussed in the introduction, images which are degraded by verylarge movements are difficult to restore. Experiments have shown that ifthe true PSF is known, a restored image can have an acceptable qualitywhere the PSF is smaller than 10% of the image size. The preferredembodiment ideally only operates on images subject to minimal movement.Thus, the original image can either be sub-sampled, preferably to ⅓ ofits original size or once the Cepstrum is computed, it can besub-sampled before further processing or indeed during furtherprocessing without having a detrimental affect on the accuracy of theestimated PSF where movement is not too severe. This can also beconsidered valid as the blurring operation may be seen as a low-passfiltering of an image (the PSF is indeed a low pass filter); andtherefore there is little benefit in looking for PSF information in thehigh frequency domain.

The next step 34 involves thresholding the negative Cepstrum. Thisassumes that only points in the negative Cepstrum with intensitieshigher than a threshold (a certain percent of the largest spike) arekept. All the other values are set to zero. This step has, also, theeffect of reducing noise. The value of the threshold was experimentallyset to 9% of the largest spike value.

Pixel candidates are then sorted with the largest spike (excluding theCepstrum center) presented first as input to a region-growing step 36,then the second spike and so on.

The region-growing step 36 has as main input a sequence of candidatepixels (referred to by location) as well as the Cepstrum and it returnsas output the number of pixels in a region around each candidate pixel.Alternatively, it could return the identities of all pixels in a regionfor counting in another step, although this is not necessary in thepresent embodiment. A region is defined as a set of points with similarCepstrum image values to the candidate pixel value. In more detail, theregion-growing step 36 operates as follows:

-   -   1. Set the candidate pixel as a current pixel.    -   2. Inspect the neighbors of the current pixel—up to 8        neighboring pixels may not already be counted in the region for        the candidate pixel or other regions. If the neighboring pixel        meets an acceptance condition, preferably that its value is        larger than 0.9 of the value of the candidate pixel value, then        include it in the region for the candidate pixel, exclude the        pixel from further regions, and increment the region size.    -   3. If a maximum number of pixels, say 128, has been reached,        exit    -   4. After finished inspecting neighbors for the current pixel, if        there are still un-investigated pixels, set the first included        pixel as the current pixel and jump to step 2.    -   5. If there are no more un-investigated adjacent pixels, exit.

As can be seen, each pixel may be included in only one region. If theregion-growing step 36 is applied to several candidate pixels, then apoint previously included in a region will be skipped when investigatingthe next regions.

After comparison of the sizes of all grown regions, step 38, the pixelchosen is the candidate pixel for the region with the greatest number ofpixels and this selected point is referred to as the PSF “end point”.The PSF “start point” is chosen the center of the Cepstrum, point 40 inFIG. 4( b)(ii).

Referring to FIG. 5( d), where the negative Cepstrum has been obtainedfrom an image, FIG. 5( e) degraded with a non-linear PSF, there areareas 46′, 46″ with spikes (rather than a single spike) which correspondto PSF turning points 48′, 48″, and it is areas such as these in normalimages which the present implementation attempts to identify inestimating the PSF for an SET image.

In a continuous space, the estimated PSF would be a straight-linesegment, such as the line 50 linking PSF start and end points, asillustrated at FIG. 6. In the present embodiment, the straight line isapproximated in the discrete space of the digital image, by pixelsadjacent the straight-line 50 linking the PSF start and end points, asillustrated at FIG. 6. Thus, all the pixels adjacent the line 50connecting the PSF start and end points are selected as being part ofthe estimated PSF, step 41. For each PSF pixel, intensity is computed byinverse proportionality with the distance from its center to the line50, step 43. After the intensities of all pixels of the PSF arecomputed, a normalization of these values is performed such that the sumof all non-zero pixels of the PSF equals 1, step 45.

Using the approach above, it has been shown that if the type of movementin acquiring the component SET images of an image is linear or nearlinear, then the estimated PSF produced by the extractor 131 asdescribed above provides good estimate of the actual PSF for deblurring.

As the curving of movement increases, during restoration, ringingproportional to the degree of curving is introduced. Similarly, ifmotion is linear but not uniform, restoration introduces ringing whichis proportional with the degree of non-uniformity. The acceptable degreeof ringing can be used to tune the motion sensor 108 and rate detector109 to produce the required quality of restored image for the leastnumber of SET images.

Also, if this PSF extractor 131 is applied to images which have beenacquired with more than linear movement, for example, night pictureshaving a long exposure time, although not useful for deblurring, theestimated PSF provided by the extractor 131 can provide a good start inthe determination of the true PSF by an iterative parametric blinddeconvolution process (not shown) for example based on MaximumLikelihood Estimation, as it is known that the results of such processesfade if a wrong starting point is chosen.

The above embodiment has been described in terms of a CMOS imagingsensor 105. In alternative implementations, a CCD image sensor or indeedany another suitable image sensor could be used. For a CCD, which istypically used with a shutter and which might normally not be consideredsuitable for providing the fine level of control required by the presentinvention, progressive readout of an image being acquired should beemployed rather than opening and closing the shutter for each SET image.

The present invention is not limited to the embodiments described aboveherein, which may be amended or modified without departing from thescope of the present invention as set forth in the appended claims, andstructural and functional equivalents thereof.

In methods that may be performed according to preferred embodimentsherein and that may have been described above and/or claimed below, theoperations have been described in selected typographical sequences.However, the sequences have been selected and so ordered fortypographical convenience and are not intended to imply any particularorder for performing the operations.

In addition, all references cited above herein, in addition to thebackground and summary of the invention sections, as well as USpublished patent applications nos. 2006/0204110, 2006/0098890,2005/0068446, 2006/0039690, and 2006/0285754, and U.S. patentapplications Nos. 60/773,714, 60/803,980, and 60/821,956, which are tobe or are assigned to the same assignee, are all hereby incorporated byreference into the detailed description of the preferred embodiments asdisclosing alternative embodiments and components.

In addition, the following United States published patent applicationsare hereby incorporated by reference for all purposes including into thedetailed description as disclosing alternative embodiments:

-   US 2005/0219391—Luminance correction using two or more captured    images of same scene.-   US 2005/0201637—Composite image with motion estimation from multiple    images in a video sequence.-   US 2005/0057687—Adjusting spatial or temporal resolution of an image    by using a space or time sequence (claims are quite broad)-   US 2005/0047672—Ben-Ezra patent application; mainly useful for    supporting art; uses a hybrid imaging system with fast and slow    detectors (fast detector used to measure PSF).-   US 2005/0019000—Supporting art on super-resolution.-   US 2006/0098237—Method and Apparatus for Initiating Subsequent    Exposures Based on a Determination of Motion Blurring Artifacts (and    2006/0098890 and 2006/0098891). The following provisional    application is also incorporated by reference: Ser. No. 60/773,714,    filed Feb. 14, 2006 entitled Image Blurring.

1. A digital image acquisition apparatus, comprising: an imageacquisition sensor coupled to imaging optics for acquiring a sequence ofimages; an image store for storing one or more of said sequence ofimages acquired by said sensor; a dedicated motion detector hardwareunit for providing hardware-based sensor control including causing saidsensor to cease capture of an image when the degree of movement of theapparatus in acquiring said image exceeds a threshold, and forselectively transferring said one or more of said images acquired bysaid sensor to said image store; and a dedicated motion extractorhardware unit for providing hardware-based determination of motionparameters of at least one a selected image stored in said image store.2. The apparatus of claim 1, wherein exposure times of two or more ofsaid sequence of images differ based on different degrees of movement ofthe apparatus.
 3. The apparatus of claim 1, further comprising adedicated image re-constructor hardware unit for providinghardware-based correction of at least one selected image with associatedmotion parameters.
 4. The apparatus of claim 3, further comprising adedicated image merger hardware unit for providing hardware-basedmerging of a selected plurality of images including said at least oneselected image corrected by said dedicated image re-constructor hardwareunit, to produce a high quality image of said scene.
 5. An apparatus asclaimed in claim 1, wherein the motion extractor is configured toestimate a point spread function (PSF) for said selected image.
 6. Anapparatus as claimed in claim 5, wherein said dedicated motion extractorhardware unit is configured to: calculate a Cepstrum for said selectedimage; identify one or more spikes in said Cepstrum; and select one ofsaid spikes in said Cepstrum as an end point for said PSF.
 7. Anapparatus as claimed in claim 6, wherein said dedicated motion extractorhardware unit is configured to calculate a negative Cepstrum, and to setpoints in said negative Cepstrum having a value less than a threshold tozero.
 8. An apparatus as claimed in claim 1, wherein said image storecomprises a temporary image store, and wherein said apparatus furthercomprises a non-volatile memory, said dedicated motion detector hardwareunit being configured to store said one or more of said images in saidnon-volatile memory.
 9. An apparatus as claimed in claim 1, wherein saiddedicated motion detector hardware unit comprises a gyro-sensor or anaccelerometer, or both.
 10. A digital image acquisition apparatus,comprising: an image acquisition sensor coupled to imaging optics foracquiring a sequence of images; an image store for storing one or moreof said sequence of images acquired by said sensor; a dedicated motiondetector hardware unit for providing hardware-based control of saidsensor to cease capture of an image when the degree of movement of theapparatus in acquiring said image exceeds a first threshold, and forcontrolling the sensor to restart capture when the degree of movement ofthe apparatus is less than a given second threshold and for selectivelytransferring said one or more of said sequence of images acquired bysaid sensor to said image store; and a dedicated motion extractorhardware unit for determining motion parameters of at least one selectedimage stored in said image store.
 11. The apparatus of claim 10, whereinexposure times of two or more of said sequence of images differ based ondifferent degrees of movement of the apparatus.
 12. The apparatus ofclaim 10, further comprising a dedicated image re-constructor hardwareunit for providing hard-based correction of at least one selected imagewith associated motion parameters.
 13. The apparatus of claim 12,further comprising a dedicated image merger hardware unit for providinghardware-based merging of a selected plurality of images including saidat least one selected image corrected by said image re-constructor, toproduce a high quality image of said scene.
 14. The apparatus as claimedin claim 10, further comprising a first exposure timer for storing anaggregate exposure time of said sequence of images, and wherein saidapparatus is configured to acquire said sequence of images until theaggregate exposure time of at least a stored number of said sequence ofimages exceeds a predetermined exposure time for said high qualityimage.
 15. An apparatus as claimed in claim 14, further comprising asecond timer for storing an exposure time for a single image, andwherein said apparatus is configured to dispose of an image having anexposure time less than a threshold time.
 16. An apparatus as claimed inclaim 10, further comprising an image quality analyzer for a singleimage, and wherein said apparatus is configured to dispose of an imagehaving a quality less than a given threshold quality.
 17. An apparatusas claimed in claim 10, wherein said image merger is configured to alignsaid selected plurality of images prior to merging said images.
 18. Anapparatus as claimed in claim 10, wherein said first and secondthresholds comprise threshold amounts of motion energy.
 19. Theapparatus of claim 10, wherein said different degrees of movementcomprise different degrees of non linear movement.
 20. The apparatus ofclaim 1, wherein said different degrees of movement comprise differentdegrees of non linear movement.
 21. The apparatus of claim 10, whereinsaid image re-constructor further for enhancing an exposure of the atleast one selected image.
 22. The apparatus of claim 1, wherein saidimage re-constructor further for enhancing an exposure of the at leastone selected image.
 23. A digital image acquisition apparatus,comprising: an image acquisition sensor coupled to imaging optics foracquiring a sequence of images; an image store for storing one or moreof said images acquired by said sensor; a dedicated motion detectorhardware unit for providing hardware-based control of said sensor tocease capture of an image when the degree of movement of the apparatusin acquiring said image exceeds a threshold; a controller forselectively transferring said one or more of said images acquired bysaid sensor to said image store; and a dedicated motion extractorhardware unit for providing hardware-based determination of motionparameters of a selected image stored in said image store;
 24. Theapparatus of claim 23, wherein the motion extractor is configured to:calculate a Cepstrum for said selected image; identify one or morespikes in said Cepstrum; and select one of said spikes in said Cepstrumas an end point for said PSF.
 25. The apparatus of claim 23, furthercomprising a dedicated image re-constructor hardware unit for providinghardware-based correction of a selected image with associated motionparameters.
 26. The apparatus of claim 25, further comprising adedicated image merger hardware unit for providing hardware-basedmerging of a selected plurality of images nominally of the same sceneand corrected by said image re-constructor to produce a high qualityimage of said scene.
 27. An apparatus as claimed in claim 23, whereinthe dedicated motion extractor hardware unit is configured to estimate apoint spread function (PSF) for said selected image.
 28. An apparatus asclaimed in claim 23, wherein said dedicated motion extractor hardwareunit is configured to calculate a negative Cepstrum, and to set pointsin said negative Cepstrum having a value less than a threshold to zero.29. An apparatus as claimed in claim 23, wherein said image storedcomprises a temporary image store, and wherein said apparatus furthercomprises a non-volatile memory, said dedicated image merger hardwareunit being configured to store said high quality image in saidnon-volatile memory.
 30. An apparatus as claimed in claim 23, whereinsaid dedicated motion detector hardware unit comprises a gyro-sensor oran accelerometer, or both.
 31. A digital image acquisition apparatus,comprising: an image acquisition sensor coupled to imaging optics foracquiring a sequence of images; an image store for storing one or moreof said images acquired by said sensor; a dedicated motion detectorhardware unit for providing hardware-based control of said sensor tocease capture of an image when the degree of movement of the apparatusin acquiring said image exceeds a first threshold; one or morecontrollers that causes the sensor to restart capture when the degree ofmovement of the apparatus is less than a given second threshold and thatselectively transfers said one or more of said images acquired by saidsensor to said image store; a dedicated motion extractor hardware unitfor providing hardware-based determination of motion parameters of aselected image stored in said image store;
 32. The apparatus of claim31, wherein the dedicated motion extractor hardware unit is configuredto: calculate a Cepstrum for said selected image; identify one or morespikes in said Cepstrum; and select one of said spikes in said Cepstrumas an end point for said PSF.
 33. The apparatus of claim 31, furthercomprising a dedicated image re-constructor hardware unit for providinghardware-based correction of a selected image with associated motionparameters.
 34. The apparatus of claim 33, further comprising adedicated image merger hardware unit for providing hardware-basedmerging of a selected plurality of images nominally of the same sceneand corrected by said dedicated image re-constructor hardware unit toproduce a high quality image of said scene.
 35. An apparatus as claimedin claim 31, further comprising a first exposure timer for storing anaggregate exposure time of said sequence of images, and wherein saidapparatus is configured to acquire said sequence of images until theaggregate exposure time of at least a stored number of said sequence ofimages exceeds a predetermined exposure time for said high qualityimage.
 36. An apparatus as claimed in claim 35, further comprising asecond timer for storing an exposure time for a single image, andwherein said apparatus is configured to dispose of an image having anexposure time less than a threshold time.
 37. An apparatus as claimed inclaim 35, further comprising an image quality analyzer for a singleimage, and wherein said apparatus is configured to dispose of an imagehaving a quality less than a given threshold quality.
 38. An apparatusas claimed in claim 31, wherein said dedicated image merger hardwareunit is configured to align said selected plurality of images prior tomerging said images.
 39. An apparatus as claimed in claim 31, whereinsaid first and second thresholds comprise threshold amounts of motionenergy.
 40. One or more dedicated hardware units for installation withina digital image acquisition apparatus having an image acquisition sensorcoupled to imaging optics for acquiring a sequence of images, and animage store for storing one or more of said sequence of images acquiredby said sensor, the one or more dedicated hardware units comprising: adedicated motion detector hardware unit for providing hardware-basedsensor control including causing said sensor to cease capture of animage when the degree of movement of the apparatus in acquiring saidimage exceeds a threshold, and for selectively transferring said one ormore of said images acquired by said sensor to said image store; and adedicated motion extractor hardware unit for providing hardware-baseddetermination of motion parameters of at least one a selected imagestored in said image store.
 41. The one or more dedicated hardware unitsof claim 40, wherein exposure times of two or more of said sequence ofimages differ based on different degrees of movement of the apparatus.42. The one or more dedicated hardware units of claim 40, furthercomprising a dedicated image re-constructor hardware unit for providinghardware-based correction of at least one selected image with associatedmotion parameters.
 43. The one or more dedicated hardware units of claim42, further comprising a dedicated image merger hardware unit forproviding hardware-based merging of a selected plurality of imagesincluding said at least one selected image corrected by said dedicatedimage re-constructor hardware unit, to produce a high quality image ofsaid scene.
 44. The one or more dedicated hardware units as claimed inclaim 40, wherein the motion extractor is configured to estimate a pointspread function (PSF) for said selected image.
 45. The one or morededicated hardware units as claimed in claim 44, wherein said dedicatedmotion extractor hardware unit is configured to: calculate a Cepstrumfor said selected image; identify one or more spikes in said Cepstrum;and select one of said spikes in said Cepstrum as an end point for saidPSF.
 46. The one or more dedicated hardware units as claimed in claim45, wherein said dedicated motion extractor hardware unit is configuredto calculate a negative Cepstrum, and to set points in said negativeCepstrum having a value less than a threshold to zero.
 47. The one ormore dedicated hardware units as claimed in claim 40, wherein said imagestore comprises a temporary image store, and wherein said apparatusfurther comprises a non-volatile memory, said dedicated motion detectorhardware unit being configured to store said one or more of said imagesin said non-volatile memory.
 48. The one or more dedicated hardwareunits as claimed in claim 40, wherein said dedicated motion detectorhardware unit comprises a gyro-sensor or an accelerometer, or both. 49.One or more dedicated hardware units for installation within a digitalimage acquisition apparatus having an image acquisition sensor coupledto imaging optics for acquiring a sequence of images, and an image storefor storing one or more of said sequence of images acquired by saidsensor, the one or more dedicated hardware units comprising: a dedicatedmotion detector hardware unit for providing hardware-based control ofsaid sensor to cease capture of an image when the degree of movement ofthe apparatus in acquiring said image exceeds a first threshold, and forcontrolling the sensor to restart capture when the degree of movement ofthe apparatus is less than a given second threshold and for selectivelytransferring said one or more of said sequence of images acquired bysaid sensor to said image store; and a dedicated motion extractorhardware unit for determining motion parameters of at least one aselected image stored in said image store.
 50. The one or more dedicatedhardware units of claim 49, wherein exposure times of two or more ofsaid sequence of images differ based on different degrees of movement ofthe apparatus.
 51. The one or more dedicated hardware units of claim 49,further comprising a dedicated image re-constructor hardware unit forproviding hard-based correction of at least one selected image withassociated motion parameters.
 52. The one or more dedicated hardwareunits of claim 51, further comprising a dedicated image merger hardwareunit for providing hardware-based merging of a selected plurality ofimages including said at least one selected image corrected by saidimage re-constructor, to produce a high quality image of said scene. 53.The one or more dedicated hardware units as claimed in claim 49, furthercomprising a first exposure timer for storing an aggregate exposure timeof said sequence of images, and wherein said apparatus is configured toacquire said sequence of images until the aggregate exposure time of atleast a stored number of said sequence of images exceeds a predeterminedexposure time for said high quality image.
 54. The one or more dedicatedhardware units as claimed in claim 53, further comprising a second timerfor storing an exposure time for a single image, and wherein saidapparatus is configured to dispose of an image having an exposure timeless than a threshold time.
 55. The one or more dedicated hardware unitsas claimed in claim 49, further comprising an image quality analyzer fora single image, and wherein said apparatus is configured to dispose ofan image having a quality less than a given threshold quality.
 56. Theone or more dedicated hardware units as claimed in claim 49, whereinsaid dedicated image merger hardware unit is configured to align saidselected plurality of images prior to merging said images.
 57. The oneor more dedicated hardware units as claimed in claim 49, wherein saidfirst and second thresholds comprise threshold amounts of motion energy.58. The one or more dedicated hardware units of claim 49, wherein saiddifferent degrees of movement comprise different degrees of non linearmovement.
 59. The one or more dedicated hardware units of claim 40,wherein said different degrees of movement comprise different degrees ofnon linear movement.
 60. The one or more dedicated hardware units ofclaim 49, wherein said dedicated image re-constructor hardware unitfurther for enhancing an exposure of the at least one selected image.61. The one or more dedicated hardware units of claim 40, wherein saiddedicated image re-constructor hardware unit further for enhancing anexposure of the at least one selected image.
 62. One or more dedicatedhardware units for installation within a digital image acquisitionapparatus having an image acquisition sensor coupled to imaging opticsfor acquiring a sequence of images, and an image store for storing oneor more of said sequence of images acquired by said sensor, the one ormore dedicated hardware units comprising: a dedicated motion detectorhardware unit for providing hardware-based control of said sensor tocease capture of an image when the degree of movement of the apparatusin acquiring said image exceeds a threshold; a controller forselectively transferring said one or more of said images acquired bysaid sensor to said image store; and a dedicated motion extractorhardware unit for providing hardware-based determination of motionparameters of a selected image stored in said image store;
 63. The oneor more dedicated hardware units of claim 62, wherein the dedicatedmotion extractor hardware unit is configured to: calculate a Cepstrumfor said selected image; identify one or more spikes in said Cepstrum;and select one of said spikes in said Cepstrum as an end point for saidPSF.
 64. The one or more dedicated hardware units of claim 62, furthercomprising a dedicated image re-constructor hardware unit for providinghardware-based correction of a selected image with associated motionparameters.
 65. The one or more dedicated hardware units of claim 64,further comprising a dedicated image merger hardware unit for providinghardware-based merging of a selected plurality of images nominally ofthe same scene and corrected by said image re-constructor to produce ahigh quality image of said scene.
 66. The one or more dedicated hardwareunits as claimed in claim 62, wherein the dedicated motion extractorhardware unit is configured to estimate a point spread function (PSF)for said selected image.
 67. The one or more dedicated hardware units asclaimed in claim 62, wherein said dedicated motion extractor hardwareunit is configured to calculate a negative Cepstrum, and to set pointsin said negative Cepstrum having a value less than a threshold to zero.68. The one or more dedicated hardware units as claimed in claim 62,wherein said image stored comprises a temporary image store, and whereinsaid apparatus further comprises a non-volatile memory, said dedicatedimage merger hardware unit being configured to store said high qualityimage in said non-volatile memory.
 69. The one or more dedicatedhardware units as claimed in claim 62, wherein said dedicated motiondetector hardware unit comprises a gyro-sensor or an accelerometer, orboth.
 70. One or more dedicated hardware units for installation within adigital image acquisition apparatus having an image acquisition sensorcoupled to imaging optics for acquiring a sequence of images, and animage store for storing one or more of said sequence of images acquiredby said sensor, the one or more dedicated hardware units comprising: adedicated motion detector hardware unit for providing hardware-basedcontrol of said sensor to cease capture of an image when the degree ofmovement of the apparatus in acquiring said image exceeds a firstthreshold; one or more controllers that causes the sensor to restartcapture when the degree of movement of the apparatus is less than agiven second threshold and that selectively transfers said one or moreof said images acquired by said sensor to said image store; a dedicatedmotion extractor hardware unit for providing hardware-baseddetermination of motion parameters of a selected image stored in saidimage store;
 71. The one or more dedicated hardware units of claim 70,wherein the dedicated motion extractor hardware unit is configured to:calculate a Cepstrum for said selected image; identify one or morespikes in said Cepstrum; and select one of said spikes in said Cepstrumas an end point for said PSF.
 72. The one or more dedicated hardwareunits of claim 70, further comprising a dedicated image re-constructorhardware unit for providing hardware-based correction of a selectedimage with associated motion parameters.
 73. The one or more dedicatedhardware units of claim 72, further comprising a dedicated image mergerhardware unit for providing hardware-based merging of a selectedplurality of images nominally of the same scene and corrected by saiddedicated image re-constructor hardware unit to produce a high qualityimage of said scene.
 74. The one or more dedicated hardware units asclaimed in claim 70, further comprising a first exposure timer forstoring an aggregate exposure time of said sequence of images, andwherein said apparatus is configured to acquire said sequence of imagesuntil the aggregate exposure time of at least a stored number of saidsequence of images exceeds a predetermined exposure time for said highquality image.
 75. The one or more dedicated hardware units as claimedin claim 74, further comprising a second timer for storing an exposuretime for a single image, and wherein said apparatus is configured todispose of an image having an exposure time less than a threshold time.76. The one or more dedicated hardware units as claimed in claim 74,further comprising an image quality analyzer for a single image, andwherein said apparatus is configured to dispose of an image having aquality less than a given threshold quality.
 77. The one or morededicated hardware units as claimed in claim 70, wherein said dedicatedimage merger hardware unit is configured to align said selectedplurality of images prior to merging said images.
 78. The one or morededicated hardware units as claimed in claim 70, wherein said first andsecond thresholds comprise threshold amounts of motion energy.